Fuzzy search using field-level deletion neighborhoods

ABSTRACT

The disclosure provides an efficient dataset search and/or deduplication that improve the speed and efficiency of dataset record search and/or deduplication over traditional methods. Certain implementations apply field-level deletion neighborhood processing to ordered field permutations of dataset records encoded with hash values. A method includes determining a field-level deletion neighborhood for two or more field combinations of the record by determining field hash values, creating field permutations, determining combined record hash values for each permutation; and associating each record hash value to the unique entity identifier. The method includes searching other entity representation records for matching combined record hash values, and assigning one or more of a unique entity identifier and a duplicate entity identifier to the other entity representation records having the matching combined record hash values. Certain implementations can include removing, from the database, at least one of the other entity representation records having a duplicate record identifier.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation application of U.S. patentapplication Ser. No. 16/904,921, filed Jun. 18, 2020, and published asU.S. Patent Application Publication No. 20210397591 on Dec. 23, 2021,the contents of which are incorporated herein as if presented in full.

BACKGROUND

The challenge of building and maintaining error-free datasets ofteninvolves searching for and removing duplicate records. The problem ofdetection and elimination of duplicate database records is one of themajor problems in the broad area of data cleansing and data quality. Asingle real-world entity may be listed multiple times in a databaseunder different records due to variations in spelling, field formats,etc. An entity listing with an address of “25 W. 6^(th) St.,” forexample, may also be duplicated in another record of the database underthe address of “25 West Sixth Street.” Such duplicate records can createunnecessary expenses, erroneous updates, and inaccurate metrics. Recorderrors are regularly introduced in real-world applications due totranscription errors, incomplete information, a lack of standardformats, etc. Such issues are discussed in the publication: “Duplicaterecord detection: A survey,” by Elmagarmid, A. K. et at., IEEETransactions on knowledge and data engineering, 19, no. 1, 2006, pages1-16.

The term “fuzzy matching” refers to a technique for finding similarwords using approximate string-matching to find partial pattern matchesbetween strings when they do not exactly match. Fuzzy matching can be aneffective method for identifying “fuzzy duplicates” that are seeminglydistinct (i.e., not exact matches) but that represent the same string.The application of fuzzy matching can be a time- and resource-intensivetask, particularly when applied to large datasets where potentiallymillions of strings need to be searched.

Conventional solutions to fuzzy matching may utilize the “LevenshteinDistance” to provide a measure of dissimilarity between strings. Theprocess of determining a Levenshtein Distance involves receiving twostrings as input and looping through the contents of the stringscharacter-by-character to determine the minimum number ofsingle-character edits (insertions, deletions, or substitutions)required to change one string into the other.

A brute-force application of the Levenshtein Distance can be used toprocess query strings, find possible misspellings, and/or suggestcorrections from a dictionary of about 50 million words within about ahalf second (assuming it takes about 10 nanoseconds to compute oneLevenshtein Distance pair), which may be viable for certainapplications, but can be too slow for certain online searchapplications.

In an example bulk matching scenario in which a new dataset containing20 million person names is to be matched to existing dataset of 100million person names (accounting for typos) a total of two quadrillioncomputations would be necessary. Even with 100 processors working inparallel (and assuming a 10 nanoseconds per Levenshtein computation),the process could take 55 hours, which is much too slow. There exists aneed for a more efficient and faster process for fuzzy matching,particularly for large datasets.

BRIEF SUMMARY

Some or all of the above needs may be addressed by certainimplementations of the disclosed technology. Certain implementations mayinclude systems and methods for fuzzy matching using deletionneighborhoods applied to fields of dataset records.

According to an example implementation, a method is provided foridentifying duplicate and near-duplicate records in a database. Thedatabase may contain a plurality of entity representation records, eachentity representation record including a unique entity identifier and aplurality of fields, each field capable of containing a field value. Foreach record of the plurality of entity representation records, themethod includes determining a field-level deletion neighborhood for twoor more field combinations of the record by: determining field hashvalues; creating field permutations; determining combined record hashvalues for each permutation; and associating each record hash value tothe unique entity identifier. For each entity representation record, themethod includes searching other entity representation records formatching combined record hash values indicating possible duplicaterecords. Certain implementations include removing, from the database, atleast one of the other entity representation records having a duplicateentity identifier.

According to an example implementation, a method is provided for findingrecords in a database using incomplete search criteria, the databasecontains a plurality of entity representation records, each entityrepresentation record including an entity identifier and a plurality offields, each field capable of containing a field value. For each recordof the plurality of entity representation records, the method includes:determining a field-level deletion neighborhood for two or more fieldcombinations of the record by: determining field hash values; creatingfield permutations; determining combined record hash values for eachpermutation; and associating each record hash value to the entityidentifier. For each entity representation record, the method includessearching other entity representation records for matching combinedrecord hash values; and assigning the unique entity identifier to theother entity representation records having the matching combined recordhash values.

According to an example implementation, a system is provided. The systemincludes a database containing a plurality of entity representationrecords, each entity representation record including a unique entityidentifier and a plurality of fields, each field capable of containing afield value; one or more processors; and memory in communication withthe one or more processors and storing instructions that, when executedby the one or more processors, cause the system to: for each record ofthe plurality of entity representation records, determine a field-leveldeletion neighborhood for two or more field combinations of the record,the field-level deletion neighborhood is determined by: computing fieldhash values; creating field permutations; computing combined record hashvalues for each permutation; and associating each record hash value tothe unique entity identifier. For each entity representation record:searching other entity representation records for matching combinedrecord hash values indicating possible duplicate records. Certainimplementations include removing, from the database, at least one of theother entity representation records having a duplicate entityidentifier.

Other implementations, features, and aspects of the disclosed technologyare described in detail herein and are considered a part of the claimeddisclosed technology. Other implementations, features, and aspects canbe understood with reference to the following detailed description,accompanying drawings, and claims.

BRIEF DESCRIPTION OF THE FIGURES

Reference will now be made to the accompanying figures and flowdiagrams, which are not necessarily drawn to scale, and wherein:

FIG. 1 is an example graphical representation of record fuzzy matchingutilizing field-level deletion neighborhoods for search and/ordeduplication, according to an example implementation of the disclosedtechnology.

FIG. 2A illustrates a string-level deletion neighborhood hash valuecreation, that may be extended to the field-level, according to certainexample implementation of the disclosed technology.

FIG. 2B illustrates hash keys generated and indexed for a single wordusing deletion neighborhood applied at the string-level, and that may beextended to the field-level deletion neighborhood, according to certainexample implementation of the disclosed technology.

FIG. 2C illustrates the use of hash keys to find matches, according tocertain example implementation of the disclosed technology.

FIG. 3A illustrates hash keys generated and indexed for a permutationsof combined record fields using deletion neighborhood applied at thefield-level, according to certain example implementation of thedisclosed technology.

FIG. 3B illustrates fuzzy matching of two separate records usingfield-level deletion neighborhood concepts, according to certain exampleimplementation of the disclosed technology.

FIG. 3C illustrates associating entity IDs for separate records havingmatching hash values using field-level deletion neighborhood concepts,according to certain example implementation of the disclosed technology.

FIG. 4 is a table listing experimental processing speed improvementsusing the deletion neighborhood concepts, according to exampleimplementations of the disclosed technology.

FIG. 5 is a block diagram of a computing device utilized in the system,in accordance with certain example implementations of the disclosedtechnology.

FIG. 6 is an example block diagram of a system that creates and/orutilizes field-level and/or string-level deletion neighborhoods forsearch and/or deduplication, according to an example implementation ofthe disclosed technology.

FIG. 7 is a flow diagram of certain methods for searching and/ordeduplicating database records, according to an example implementationof the disclosed technology.

DETAILED DESCRIPTION

The disclosed technology generally relates to dataset search and/ordeduplication, and more particularly, to systems and methods thatimprove the speed and efficiency of dataset record search and/ordeduplication over traditional methods. Certain implementations applydeletion neighborhood processing to ordered field permutations ofdataset records encoded with hash values. The disclosed technology cansignificantly reduce the candidate pool size for matching records byextending traditional character- and word-level techniques tomulti-field- and record-level fuzzy matching, which can provide certaintechnical benefits and advantages such as increased speed and reducedruntime.

Certain implementations of the disclosed technology utilize field-leveldeletion neighborhoods, field group combinations, hashing of individualfield groups, and/or hashing of combined field groups. The generalconcept of deletion neighborhoods, as applied to the character-leveledit distance of words in dictionaries, is discussed in “Fast similaritysearch in large dictionaries,” Bocek, Thomas, et al., University, 2007;and in “Improved fast similarity search in dictionaries,” Karch, Daniel,et al., International Symposium on String Processing and InformationRetrieval. Springer, Berlin, Heidelberg, 2010.

Certain implementations of the disclosed technology will be describedmore fully hereinafter with reference to the accompanying drawings. Thisdisclosed technology may, however, be embodied in many different formsand should not be construed as limited to the implementations set forthherein.

FIG. 1 is an example graphical representation 100 of record fuzzymatching that utilizes field-level deletion neighborhoods for searchand/or deduplication, according to an example implementation of thedisclosed technology. For simplicity, this example shows arepresentation of four database records (labeled A, B, C, D), eachhaving a plurality of associated fields. By way of an example, a firstentity record may be represented by a unique entity identifier “A” 102,and the record may include associated fields with designators such as:First Name, Last Name, Address Number, Address Street, Phone Number,City, and Zip code. The other records of the database (B, C, and D) mayinclude the same fields and fields designators and may be populated withfield entries corresponding to the particular entity.

A deletion neighborhood 104 corresponding to the first entity A 102 maybe created by sequentially “deleting” field entries then combining theremaining field entries to form different variations of the record, eachhaving certain fields or field combinations de-emphasized. This processessentially enables finding records that match without requiring allfield values to match exactly. The full record for entity A 102, forexample, may include field entries for each of the associated fields,such as John, Smith, 123, Main Street, 123456789, Dallas, 75001, whilethe first entry A1 106 in the deletion neighborhood 104 for entity A 102may omit the first name (Smith, 123, Main Street, 123456789, Dallas,75001). The second entry A2 in the deletion neighborhood 104 for entityA 102 may omit the last name (John, 123, Main Street, 123456789, Dallas,75001), and so forth for each deletion neighborhood 110, 114, 116corresponding to the other entities B, C, and D.

As will be discussed below, a field hash value may be computed for eachfield value, and each entry or member (e.g., A1, A2, A3, A4, A5, A6) ofthe corresponding deletion neighborhood (e.g., 104) may be representedby a combined hash value of the remaining ordered field values.

Once the deletion neighborhoods 104, 110, 114, 116 are determined forthe corresponding entities, (A, B, C, D), the database can be searchedfor partially matching records in the other deletion neighborhoods. Forexample, the graphical representation 100 of FIG. 1 depicts a scenariowhere the third member A3 108 of the deletion neighborhood 104 of entityA 102 matches with the sixth member B6 108 of the deletion neighborhood110 of the entity B. Additionally, as depicted, the graphicalrepresentation 100 of FIG. 1, the fourth member B4 112 of the deletionneighborhood 110 of entity B matches with the first member D1 112 of thedeletion neighborhood 114 of the entity D. In this example, the entity A102 differs from entity B by a field edit distance of 1, and entity Ddiffers from entity B by a field edit distance of 1.

According to certain example implementations of the disclosedtechnology, the deletion neighborhood process can be further applied tosuppress additional fields to enable matches to be found for recordshaving edit distances greater than 1. This scenario is represented inthe double circle representation 118 in which sub-member A55 120 of adeletion neighborhood based on member A5 of the first deletionneighborhood 104 matches sub-member C15 122 of a deletion neighborhoodbased on member Cl of the fourth deletion neighborhood 116. In thisexample scenario, A55 120 may match C15 122 with a field edit distanceof 2, meaning that all but up to two fields match in the recordscorresponding to entity A and C. In certain aspects, the “edit distance”in relation to deletion neighborhoods, can be considered a “maximum editdistance” for the purpose of creating the entries for the neighborhood.For example, when edit distance=1, entries may be generated where editdistance=0 or 1; when edit distance=2, entries may be generated whereedit distance=0, 1, and 2. An edit distance=0 implies an exact matchcase. Thus, when edit distance=2, entities can match (or may beconsidered to be in the neighborhood) when up to 2 fields mismatch. Inaccordance with certain example implementations of the disclosedtechnology, various levels of deletion neighborhoods may be utilized toenhance the performance of the fuzzy matching process. Certainimplementations may utilize field-level deletion neighborhood(s), fieldvalue-level deletion neighborhood(s), and/or a combination thereof.

FIGS. 2A, 2B, and 2C will now be discussed to illustrate the concepts ofthe deletion neighborhood and hash values applied at the string level(or field value-level). FIGS. 3A, 3B, and 3C will be used to illustratethe application of the deletion neighborhood concepts and hashing to thefield level, in accordance with certain example implementations of thedisclosed technology.

FIG. 2A depicts a creation of a deletion neighborhood (with editdistance=1) and associated hash values corresponding to a dictionaryword “LEAST” and the associated fragments of the word that are createdby sequentially deleting one letter at a time. In this example, a hashfunction takes the dictionary word and its fragments as key inputs andproduces corresponding hash values as output. In certain exampleimplementations, the hash function may convert variable length inputkeys into fixed length hash values. Various hash functions and the usesthereof are discussed in Wang, et al., “Hashing for Similarity Search: ASurvey,” 2014, 1408.2927, arXiv.

FIG. 2B depicts an example of query processing, including the creationof a system index where each of the hash values (computed with referenceto FIG. 2A) are used as hash keys and assigned to the same dictionaryword “LEAST.” When a user attempts to use a keyboard to enter the word“LEAST” for a query, it is quite easy to mistakenly hit the “Q” keyinstead of the “A” key on a QWERTY keyboard due to the proximity ofthese keys. In this example, such a common typing mistake (with an editdistance of 1) may be compensated for by determining a correspondingdeletion neighborhood for the word “LEQST.” FIG. 2B depicts the creationof a deletion neighborhood for “LEQST” and it fragments, including thegeneration of associated hash values. In certain implementations, aquery's deletion neighborhood may be computed at runtime, then joinedagainst a pre-built system index.

FIG. 2C depicts a match between a hash value for the fragment of “LEQST”in which the “Q” character is suppressed, and a hash key for thefragment of “LEAST” in which the “A” character is suppressed, whichenables finding a match of the hash values that correspond to theremaining characters “LE_ST.” The examples depicted in FIG. 2A-2Ccorrespond to a Levenshtein distance of 1, which is the minimum numberof single-character edits (insertions, deletions, or substitutions)required to change one word into another. In deletion neighborhoodsearching (i.e., matching the hash values) a list of candidates may beproduced that can then be verified. In the example case where editdistance=1 and the dictionary contains the word “skit,” a query word of“kits” will match with edit distance=1 on the fragment “kit.” However,using an insert/delete/substitute rule, the two words would beconsidered to have an edit distance=2.

Some of the general concepts of string-level deletion neighborhoodgeneration, as depicted and discussed with reference to FIGS. 2A-C, maybe utilized in understanding the novel extension to the process offield-level deletion neighborhood generation, as will now be discussedwith reference to FIGS. 3A-C. In the string-level deletion neighborhoodgeneration, the following concepts may apply: (1) a single word may beconsidered as an item of information; (2) a word is composed of orderedcharacters; (3) a character is a code value; and (4) an operation thatcombines characters into a word is a left-to-right concatenation. Incontrast, the following concepts may apply to field-level deletionneighborhood generation: (1) a single record in a dataset may beconsidered as an item of information; (2) a record is composed ofordered field values; (3) a code value is a hash value of a field value;and (4) an operation that combines code values is a left-to-righthashing of hashed field values.

FIG. 3A is an example table 302 depicting field-level deletionneighborhood hash value entries 306 and field combination hash keys 310that may be created from a database record 304 having a plurality offields (e.g., First Name, Last Name, Street Num, Street Name,City+State) populated with field values (e.g., John, Smith, 123, MainSt., Austin+Texas) in accordance with an example implementations of thedisclosed technology. For clarity, FIG. 3A also depicts each of thefield permutations 308 used in computing the field combination hash keys310. The database record 304 may be associated with an entity identifierID 312, which in certain implementations, may also be associated withthe field combination hash keys 310, as will be discussed below withreference to FIG. 3B.

In certain example implementations, requiring city/state/zip codeinformation in the deletion neighborhood generation may improve theaccuracy of the search. In some implementations, certain fields may becombined to further improve the search by providing context to thesearch. The process of combining fields may also provide the benefit ofreducing memory requirements to store the generated records. FIG. 3Aillustrates the process of combining certain fields (such as City+State)before computing the field hash value entries 306, which may then beused to compute the field combination hash keys 310. In certainimplementations, the street name (e.g., MAIN) and the street name suffix(e.g., St) may be similarly combined before computing the field hashvalue entries.

FIG. 3B illustrates fuzzy matching a first separate record 314 and asecond separate record 316 to the database record 304 as shown anddiscussed above with reference to FIG. 3A using field-level deletionneighborhood concepts, according to certain example implementation ofthe disclosed technology. In this example, a field-level deletionneighborhood is generated for the first separate record 314, and aresulting field combination hash key 320 (generated by deletion of the“First Name” field entry 318) matches a generated field combination hashkey 310 corresponding to the database record 304. Given the match, theID 312 corresponding to the database record 304 may be associated withthe first separate record 314. In certain example implementations, afield edit distance (1 in this case) may also be associated with theconnection between the database record 304 and the first separate record314 since all but one of the fields match.

FIG. 3B also depicts a field combination hash key 320 (generated bydeletion of the “Street Num” field entry 332 in the second separaterecord 316) that matches a generated field combination hash key 324corresponding to the database record 304. Given the match, the ID 312corresponding to the database record 304 may also be associated with thesecond separate record 316. In certain example implementations, a fieldedit distance (1 in this case) may also be associated with theconnection between the database record 304 and the second separaterecord 316 since all but one of the fields match.

FIG. 3C summarizes the results of the fuzzy record matching examplediscussed with reference to FIG. 3A and FIG. 3B. According to an exampleimplementation, entity IDs 312 may be linked 326 for those databaserecords 330 having matching field combination hash keys 310.

FIG. 4 is a table 400 listing experimental processing speed improvementsusing the deletion neighborhood concepts for single searches and bulkmatching compared with brute-force methods. Significant advantages inprocessing speed may be realized with the disclosed technology,particularly when applied to large datasets. For example, processingtime may be reduced by a factor of 388 (over force matching) for bulkmatching 73,000 search fields in a dataset containing 370,000 recordsutilizing embodiments of the disclosed technology.

In accordance with certain example implementations of the disclosedtechnology, various levels (record-level, field-level, fieldvalue-level, string-level etc.) of deletion neighborhoods may beutilized in the process of fuzzy matching for search and/ordeduplication. And as illustrated in FIG. 1, the fuzzy matchingtechniques disclosed herein may be applied to find fuzzy matches withedit distances (character, string, field, etc.) of any value. In certainexample implementations, it may be preferred to limit the edit distancecalculations to 5 or less, for example, to reduce false positive matchesand/or to limit the resulting memory requirements to store the results.Certain example implementations of the disclosed technology may utilizeother types of neighborhoods besides or in addition to deletionneighborhoods, including but not limited to fumble-finger neighborhoods,character transposition neighborhoods, alternate representationneighborhoods. A fumble-finger neighborhood, for example, may beconstructed using versions of field values that are common typographicalerrors. A character transposition neighborhood, for example, may beconstructed using versions of field values with adjacent characterstransposed. An alternate representation neighborhood, for example, maybe constructed using versions of field values with synonyms, nicknames,Soundex, and/or Metaphonic representations.

The number of records N that may be generated in the process ofgenerating field-level deletion neighborhoods may be estimated accordingto the formula:

${N = {m\left( {\sum\limits_{r = 1}^{a}\frac{n!}{{r!}{\left( {n - r} \right)!}}} \right)}},$

where n=the total number of field groups, r=the edit distance, a=themaximum edit distance, and m=the number of records. If deletionneighborhoods are also generated at the string-level, the value n is theaverage length of the field values. If string-level deletionneighborhoods are also computed, the calculation above would beperformed for every field that is expanded with string-level deletionneighborhoods and, at each step, the previous N result becomes the m forthe next calculation. As an example using only field-level deletionneighborhoods, a dataset having a number of records m=363.8 million with5 field groups would expand to approximately 2.2 billion records uponcreating a deletion neighborhood for five field groups with an editdistance of 1. In this example, the result of the deletion neighborhoodcreation (in which an entity ID can be converted to a hash value, and ahash value can be converted to an entity ID), the size of each the twodata indexes for this example would be about 31 GB in size, with a totalsize of the indexes approximately 62 GB. This example is representativeof the approximate size of data that may be generated using a publicrecord header file with fully populated names, SSN, and addresses.Example case examples are discussed at the end of this document. Theindex size values discussed above were determined experimentally usingHPCC Systems technology; other technologies may produce differentresults.

The disclosed technology may enable fuzzy deduplication of a dataset(such as a public records header) that was previously unachievable oncertain computing platforms due to the size and complexity of the data.Certain example implementations of the disclosed technology may beenabled by the use of special purpose HPCC Systems in combination withspecial purpose software. HPCC Systems, for example, are developed andoffered by LexisNexis Risk Solutions, Inc., the assignee of thedisclosed technology and provide data-intensive supercomputingplatform(s) designed for solving big data problems. Certain softwaremodules, in conjunction with the HPCC Systems, provide technicalimprovements in computer processing that enable the disclosed technologyand provide useful, tangible results that may have previously beenunattainable. For example, certain example implementation of thedisclosed technology may process massive data sets, which arecomputationally intensive, requiring special software and hardware.

One of the issues that has plagued previous “relationship determination”solutions involving massive data sets is the extremely long run-timesand large amount of memory/disk space required. One of the technicalsolutions provided by the technology disclosed herein concerns theenablement and efficiency improvement of computer systems and softwareto process relationship data, and to provide the desired data in areasonable amount of time.

Determining relationships among records, for example, can follow theclassical n-squared process for both time and disk space. According toan example implementation of the disclosed technology, light-weightself-joins may be utilized, for example, in generating EnterpriseControl Language (ECL). But disk-space utilization might still be high.Certain example implementations of the disclosed technology may enable acore join to be split into parts, each of which is persisted. This hasthe advantage of breaking a potentially very long join into n partswhile allowing others a time slice. This has an effect of reducing diskconsumption by a factor of n, provided the eventual links are fairlysparse. In terms of performance, it should be noted that if n can bemade high enough that the output of each join does not spill to disk,the relationship calculation process may have significantly fasterperformance.

In accordance with certain example implementations, deduplicationrecords may be performed by certain additional special programming andanalysis software. For example, record linking fits into a general classof data processing known as data integration, which can be defined asthe problem of combining information from multiple heterogeneous datasources. Data integration can include data preparation steps such asparsing, profiling, cleansing, normalization, and parsing andstandardization of the raw input data prior to record linkage to improvethe quality of the input data and to make the data more consistent andcomparable (these data preparation steps are sometimes referred to asETL or extract, transform, load).

Certain example implementations of the disclosed technology may utilizedata profiling and data hygiene applications to support the datapreparation process. For example, a data ingest application may allowinput files to be combined or merged with an existing base file. Incertain implementations, unstructured data may be parsed for datapreparation. The data preparation steps are usually followed by theactual record linking or clustering process. Certain implementations ofthe disclosed technology may utilize applications for several differenttypes of record linking.

Data profiling, data hygiene and data source consistency checking, whilekey components of the record linking process, have their own valuewithin the data integration process and may be supported for leverageeven when record linking is not a necessary part of a particular datawork unit. Certain example implementations may utilize specificity todetermine the relevance/weight of a particular field in the scope of thelinking process, and a mathematical model based on the input data,rather than the need for hand coded user rules, which may be key to theoverall efficiency of the method.

Certain implementations of the disclosed technology may be structured aslibrary function. In some implementations, automatic deduplication maybe performed based on the result of the search. In other exampleimplementations, a user may decide what to do with the results. In oneexemplary implementation of the disclosed technology, a user may receivenew data and may want to determine if any of the new data was alreadypresent in the database. In certain implementations, the search resultsmay be fed into a deduplicator. According to certain exemplaryimplementations of the disclosed technology, the deduplicator may beconfigured to perform one or more of the following:

-   -   If the new data is known to be a complete refresh (i.e. all        fields complete) and is highly trusted, the deduplicator may        assign the found entity ID to the new data and delete the old        data.    -   If the new data is partial but highly trusted, the old data may        be merged into the new data while prioritizing new field values.        If the entity ID is found in the old data, it may be added to        the new data and the old data may be deleted.    -   If the new data is not highly trusted, the old and new data may        be merged while prioritizing the old data and re-writing the old        data.

In certain implementations, the merging can be performed on afield-by-field basis. For example, old and new field values may becompared, and the “better” field value may be retained. In someimplementations a better field value may be based on a number offactors, including but not limited to confidence scores, specificity,etc. Examples of better fields populated by name information couldinclude fields having the longest names for first names and/ornon-nicknames. Examples of better fields populated by latitude/longitudeinformation could include fields having the greatest decimal pointaccuracy. Examples of better fields populated by date information couldinclude those fields with full date information (“17/Feb/1965”) ratherthan partial date information (“Feb/1965”).

In certain example implementations, merely identifying duplicates may besufficient and valuable. As an example, consider the case where allrecords may have valid beginning and ending dates. If a duplicate isfound, a detailed comparison of the data may be done. If the resultshows that the records are essentially the same, the old record's enddate may be updated to the current date. If the records are differentenough, a new record may be created, possibly using merge rules. In someimplementation, the new record can be assigned with the found entity ID,the beginning and ending dates may be updated to the current date, andthe overall result may be appended to the old dataset. In this respect,record history may be preserved, and subsequent queries of the data canbe made as of a certain date.

In certain example implementations, field groups may be compared insteadof individual fields, particularly in cases where individual fieldvalues provide little meaning or specificity absent a connection toother field values.

Certain example implementations may return only entity IDs as results.In such implementations, the original data may not be retained at all.Thus, in certain aspects, the original data may be considered anonymizedas hash codes but may still be searchable, which may provide certainadvantages for applications such as HIPAA (among others) wherepersonally identifiable information (PII) needs to be removed from datato avoid privacy issues. In this respect, a user may be required toseparately fetch the original data for display, deduplication, orfurther use.

FIG. 5 depicts a block diagram of an illustrative computing device 500that may be utilized for efficient record search and/or recorddeduplication according to an example implementation. Variousimplementations and methods herein may be embodied in non-transitorycomputer readable media for execution by a processor. It will beunderstood that the computing device 500 is provided for examplepurposes only and does not limit the scope of the variousimplementations of the communication systems and methods.

The computing device 500 of FIG. 5 includes one or more processors wherecomputer instructions are processed. The computing device 500 maycomprise the processor 502, or it may be combined with one or moreadditional components shown in FIG. 5. In some instances, a computingdevice may be a processor, controller, or a central processing unit(CPU). In yet other instances, a computing device may be a set ofhardware components.

The computing device 500 may include a display interface 504 that actsas a communication interface and provides functions for rendering video,graphics, images, and texts on the display. In certain exampleimplementations of the disclosed technology, the display interface 504may be directly connected to a local display. In another exampleimplementation, the display interface 504 may be configured forproviding data, images, and other information for an external/remotedisplay. In certain example implementations, the display interface 504may wirelessly communicate, for example, via a Wi-Fi channel or otheravailable network connection interface 512 to the external/remotedisplay.

In an example implementation, the network connection interface 512 maybe configured as a communication interface and may provide functions forrendering video, graphics, images, text, other information, or anycombination thereof on the display. In one example, a communicationinterface may include a serial port, a parallel port, a general purposeinput and output (GPIO) port, a game port, a universal serial bus (USB),a micro-USB port, a high definition multimedia (HDMI) port, a videoport, an audio port, a Bluetooth port, a near-field communication (NFC)port, another like communication interface, or any combination thereof.In one example, the display interface 504 may be operatively coupled toa local display. In another example, the display interface 504 maywirelessly communicate, for example, via the network connectioninterface 512 such as a Wi-Fi transceiver to the external/remotedisplay.

The computing device 500 may include a keyboard interface 506 thatprovides a communication interface to a keyboard. According to certainexample implementations of the disclosed technology, thepresence-sensitive display interface 508 may provide a communicationinterface to various devices such as a pointing device, a touch screen,etc.

The computing device 500 may be configured to use an input device viaone or more of input/output interfaces (for example, the keyboardinterface 506, the display interface 504, the presence sensitive displayinterface 508, network connection interface 512, camera interface 514,sound interface 516, etc.,) to allow a user to capture information intothe computing device 500. The input device may include a mouse, atrackball, a directional pad, a track pad, a touch-verified track pad, apresence-sensitive track pad, a presence-sensitive display, a scrollwheel, a digital camera, a digital video camera, a web camera, amicrophone, a sensor, a smartcard, and the like. Additionally, the inputdevice may be integrated with the computing device 500 or may be aseparate device. For example, the input device may be an accelerometer,a magnetometer, a digital camera, a microphone, and an optical sensor.

Example implementations of the computing device 500 may include anantenna interface 510 that provides a communication interface to anantenna; a network connection interface 512 that provides acommunication interface to a network. According to certain exampleimplementations, the antenna interface 510 may utilize to communicatewith a Bluetooth transceiver.

In certain implementations, a camera interface 514 may be provided thatacts as a communication interface and provides functions for capturingdigital images from a camera. In certain implementations, a soundinterface 516 is provided as a communication interface for convertingsound into electrical signals using a microphone and for convertingelectrical signals into sound using a speaker. According to exampleimplementations, random-access memory (RAM) 518 is provided, wherecomputer instructions and data may be stored in a volatile memory devicefor processing by the CPU 502.

According to an example implementation, the computing device 500includes a read-only memory (ROM) 520 where invariant low-level systemcode or data for basic system functions such as basic input and output(I/O), startup, or reception of keystrokes from a keyboard are stored ina non-volatile memory device. According to an example implementation,the computing device 500 includes a storage medium 522 or other suitabletype of memory (e.g. such as RAM, ROM, programmable read-only memory(PROM), erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), magnetic disks, opticaldisks, floppy disks, hard disks, removable cartridges, flash drives),where the files include an operating system 524, application programs526 (including, for example, a web browser application, a widget orgadget engine, and or other applications, as necessary) and data files528 are stored. According to an example implementation, the computingdevice 500 includes a power source 530 that provides an appropriatealternating current (AC) or direct current (DC) to power components.According to an example implementation, the computing device 500includes and a telephony subsystem 532 that allows the device 500 totransmit and receive sound over a telephone network. The constituentdevices and the CPU 502 communicate with each other over a bus 534.

In accordance with an example implementation, the CPU 502 hasappropriate structure to be a computer processor. In one arrangement,the computer CPU 502 may include more than one processing unit. The RAM518 interfaces with the computer bus 534 to provide quick RAM storage tothe CPU 502 during the execution of software programs such as theoperating system application programs, and device drivers. Morespecifically, the CPU 502 loads computer-executable process steps fromthe storage medium 522 or other media into a field of the RAM 518 inorder to execute software programs. Data may be stored in the RAM 518,where the data may be accessed by the computer CPU 502 during execution.In one example configuration, the device 500 includes at least 128 MB ofRAM, and 256 MB of flash memory.

The storage medium 522 itself may include a number of physical driveunits, such as a redundant array of independent disks (RAID), a floppydisk drive, a flash memory, a USB flash drive, an external hard diskdrive, thumb drive, pen drive, key drive, a High-Density DigitalVersatile Disc (HD-DVD) optical disc drive, an internal hard disk drive,a Blu-Ray optical disc drive, or a Holographic Digital Data Storage(HDDS) optical disc drive, an external mini-dual in-line memory module(DIMM) synchronous dynamic random access memory (SDRAM), or an externalmicro-DIMM SDRAM. Such computer readable storage media allow the device500 to access computer-executable process steps, application programsand the like, stored on removable and non-removable memory media, tooff-load data from the device 500 or to upload data onto the device 500.A computer program product, such as one utilizing a communication systemmay be tangibly embodied in storage medium 522, which may comprise amachine-readable storage medium.

According to one example implementation, the term computing device, asused herein, may be a CPU, or conceptualized as a CPU (for example, theCPU 502 of FIG. 5). In this example implementation, the computing device(CPU) may be coupled, connected, and/or in communication with one ormore peripheral devices.

FIG. 6 is an example block diagram of a system 600 that creates and/orutilizes field-level and/or string-level deletion neighborhoods forsearch and/or deduplication, according to an example implementation ofthe disclosed technology. The system 600 may include (or be embodied as)one or more of the computing device 500 components as discussed abovewith reference to FIG. 6. The system 600 may be embodied as a specialpurpose HPCC system 610 as previously discussed, and may utilize one ormore a special-purpose modules, including but not limited to: afield-level deletion neighborhood generator 612, a hash value generator614, a hash search module 616, a string-level deletion neighborhoodgenerator, and de-duplicator module 620.

In certain example implementations, the HPCC system 610 may be incommunication with one or more local database(s) 630. In certain exampleimplementations, the HPCC system 610 may be in communication with aserver 650, for example via a local area network 640. In certainimplementations, the server may be in communication to a wide areanetwork 660 (such as the Internet) to enable remote access to the HPCCsystem 610. In certain implementations, one or more remote database(s)670 may be in communication with the wide area network 670, for example,to store data and/or to retain processed data.

An example method 700 will now be described with reference to theflowchart of FIG. 7. The method may include searching for and/orde-duplicating records in a database containing a plurality of entityrepresentation records, each entity representation record including aunique entity identifier and a plurality of fields, each field capableof containing a field value. For each record of the plurality of entityrepresentation records, the method 700 includes determining afield-level deletion neighborhood for two or more field combinations ofthe record. The method 700 starts in block 702, and according to anexample implementation includes determining field hash values. In block704, the method 700 includes creating field permutations. In block 706,the method 700 includes determining combined record hash values for eachpermutation. In block 708, the method 700 includes associating eachrecord hash value to the unique entity identifier. In block 710, and foreach entity representation record the method 700 includes searchingother entity representation records for matching combined record hashvalues. In block 712, the method 700 includes assigning one or more of aunique entity identifier and a duplicate entity identifier to the otherentity representation records having the matching combined record hashvalues.

In an example implementation, the method can further include removing,from the database, at least one of the other entity representationrecords having the duplicate entity identifier.

The method can further include saving at least one unique field value ofthe removed entity representation records in a data preserving fieldidentified by at least the unique entity identifier.

The method can further include removing, from the database, all but onerepresentative entity representation record from records having a sameassigned unique entity identifier. In certain example implementations,the representative entity representation record can include a recordhaving a highest accuracy confidence level.

In certain example implementations, the two or more field combinationsof the record can include one of more of: predetermined fields, requiredfields, combined fields, and ordered fields. In some implementations,the required fields can include city and state fields.

The method can further include determining a field string-level deletionneighborhood for each field value.

Throughout the specification and the claims, the following terms take atleast the meanings explicitly associated herein, unless the contextclearly dictates otherwise. The term “connected” means that onefunction, feature, structure, or characteristic is directly joined to orin communication with another function, feature, structure, orcharacteristic. The term “coupled” means that one function, feature,structure, or characteristic is directly or indirectly joined to or incommunication with another function, feature, structure, orcharacteristic. The term “or” is intended to mean an inclusive “or.”Further, the terms “a,” “an,” and “the” are intended to mean one or moreunless specified otherwise or clear from the context to be directed to asingular form.

Throughout the specification and the claims, numerous specific detailsare set forth. However, it is to be understood that implementations ofthe disclosed technology may be practiced without these specificdetails. In other instances, well-known methods, structures andtechniques have not been shown in detail in order not to obscure anunderstanding of this description. References to “one implementation,”“an implementation,” “example implementation,” “variousimplementations,” etc., indicate that the implementation(s) of thedisclosed technology so described may include a particular feature,structure, or characteristic, but not every implementation necessarilyincludes the particular feature, structure, or characteristic. Further,repeated use of the phrase “in one implementation” does not necessarilyrefer to the same implementation, although it may.

The various aspects described herein are presented as methods, devices(or apparatus), systems, and articles of manufacture that may include anumber of components, elements, members, modules, nodes, peripherals, orthe like. Further, these methods, devices, systems, and articles ofmanufacture may include or not include additional components, elements,members, modules, nodes, peripherals, or the like.

According to one example implementation, the terms computing device, asused herein, may be a CPU, or conceptualized as a CPU (for example, theCPU 502 of FIG. 5). In certain example implementations, the computingdevice (CPU) may be coupled, connected, and/or in communication with oneor more peripheral devices. In another example implementation, the termcomputing device, as used herein, may refer to a handheld computer,desktop computer, laptop computer, tablet computer, or some other liketerminology. In an example embodiment, the mobile computing device mayoutput content to its local display and/or speaker(s).

Furthermore, the various aspects described herein may be implementedusing special-purpose programming or engineering techniques to producesoftware, firmware, hardware, or any combination thereof to control acomputing device to implement the disclosed subject matter. The term“system” as used herein is intended to encompass a computer programaccessible from any computing device, carrier, or media. For example, acomputer-readable medium may include: a magnetic storage device such asa hard disk, a floppy disk or a magnetic strip; an optical disk such asa compact disk (CD) or digital versatile disk (DVD); a smart card; and aflash memory device such as a card, stick or key drive. Additionally, itshould be appreciated that a carrier wave may be employed to carrycomputer-readable electronic data including those used in transmittingand receiving electronic data such as electronic mail (e-mail) or inaccessing a computer network such as the Internet or a local areanetwork (LAN). Of course, a person of ordinary skill in the art willrecognize many modifications may be made to this configuration withoutdeparting from the scope or spirit of the claimed subject matter.

As used herein, unless otherwise specified the use of the ordinaladjectives “first,” “second,” “third,” etc., to describe a commonobject, merely indicate that different instances of like objects arebeing referred to, and are not intended to imply that the objects sodescribed must be in a given sequence, either temporally, spatially, inranking, or in any other manner.

One or more network interfaces may facilitate connection of thecomputing device 500 inputs and outputs to one or more suitable networksand/or connections. The one or more network interfaces may furtherfacilitate connection to one or more suitable networks; for example, alocal area network, a wide area network, the Internet, a cellularnetwork, a radio frequency network, a Bluetooth enabled network, a Wi-Fienabled network, a satellite-based network any wired network, anywireless network, etc., for communication with external devices and/orsystems.

As desired, implementations of the disclosed technology may include thecomputing device 500 with more or less of the components illustrated inFIG. 5.

Certain implementations of the disclosed technology are described abovewith reference to block and flow diagrams of systems and methods and/orcomputer program products according to example implementations of thedisclosed technology. It will be understood that one or more blocks ofthe block diagrams and flow diagrams, and combinations of blocks in theblock diagrams and flow diagrams, respectively, can be implemented bycomputer-executable program instructions. Likewise, some blocks of theblock diagrams and flow diagrams may not necessarily need to beperformed in the order presented or may not necessarily need to beperformed at all, according to some implementations of the disclosedtechnology.

These computer-executable program instructions may be loaded onto ageneral-purpose computer, a special-purpose computer, a processor, orother programmable data processing apparatus to produce a particularmachine, such that the instructions that execute on the computer,processor, or other programmable data processing apparatus create meansfor implementing one or more functions specified in the flow diagramblock or blocks. These computer program instructions may also be storedin a computer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meansthat implement one or more functions specified in the flow diagram blockor blocks. As an example, implementations of the disclosed technologymay provide for a computer program product, comprising a computer-usablemedium having a computer-readable program code or program instructionsembodied therein, said computer-readable program code adapted to beexecuted to implement one or more functions specified in the flowdiagram block or blocks. The computer program instructions may also beloaded onto a computer or other programmable data processing apparatusto cause a series of operational elements or steps to be performed onthe computer or other programmable apparatus to produce acomputer-implemented process such that the instructions that execute onthe computer or other programmable apparatus provide elements or stepsfor implementing the functions specified in the flow diagram block orblocks.

Accordingly, blocks of the block diagrams and flow diagrams supportcombinations of means for performing the specified functions,combinations of elements or steps for performing the specified functionsand program instruction means for performing the specified functions. Itwill also be understood that each block of the block diagrams and flowdiagrams, and combinations of blocks in the block diagrams and flowdiagrams, can be implemented by special-purpose, hardware-based computersystems that perform the specified functions, elements or steps, orcombinations of special-purpose hardware and computer instructions.

While certain implementations of the disclosed technology have beendescribed in connection with what is presently considered to be the mostpractical and various implementations, it is to be understood that thedisclosed technology is not to be limited to the disclosedimplementations, but on the contrary, is intended to cover variousmodifications and equivalent arrangements included within the scope ofthe appended claims. Although specific terms are employed herein, theyare used in a generic and descriptive sense only and not for purposes oflimitation.

This written description uses examples to disclose certainimplementations of the disclosed technology, including the best mode,and also to enable any person skilled in the art to practice certainimplementations of the disclosed technology, including making and usingany devices or systems and performing any incorporated methods. Thepatentable scope of certain implementations of the disclosed technologyis defined in the claims, and may include other examples that occur tothose skilled in the art. Such other examples are intended to be withinthe scope of the claims if they have structural elements that do notdiffer from the literal language of the claims, or if they includeequivalent structural elements with insubstantial differences from theliteral language of the claims.

Use Case 1 Fuzzy Deduplication

The disclosed technology may be utilized to deduplicate records in alarge dataset, where each record is identified by a unique entity ID,but where some of the records have nearly identical data. The dataset,for example, may be a public records header with about 500 millionrecords populated with names, SSNs, addresses, etc. In this example,certain fields may be considered (such as: first name, last name, SSN,street address number+street address name, city name+zip code). Certainfields, such as the street address number and the street address name,may be combined into a single field group (as indicated by the “+”),particularly when no appreciable benefit in processing speed, memoryutilization, and/or accuracy is gained by separately generating deletionneighborhoods for such fields. Furthermore, certain fields (such as thefirst name) may be expanded with an edit distance=1 to account formisspellings, etc.

In accordance with certain implementations of the disclosed technology,a first pair of ID indexes may be generated, for example, to keep trackof the generated deletion neighborhoods and the associated records.

To further help eliminate self-join scenarios and/or to avoidunnecessarily producing additional records, certain implementation mayinclude creating another pair of system indexes based on informationsupplied from the first pair of ID indexes. This process may includefinding initial matches (via hash codes), creating entity groups withthose initial matches, identifying the initial matches with a new uniqueID, and recursively remapping that new ID to its lowest value.

Experimental results using the techniques disclosed herein on a publicrecords header containing 453,705,083 records found 34.7 millionclusters of near duplicates, with clusters ranging in size from 2 to1157. It was determined that less than 1% of the clusters included falsepositive (i.e., records incorrectly designated as duplicate). Bycompletely deduplicating (resolving) the clusters, approximately 45million duplicate records could be removed from the public recordsheader.

Use Case 2 Fuzzy Form Search

The disclosed technology may be utilized for increasing processingspeeds related to form searching. In this use case, a user may fill-outone or more fields of a search form to find matching records (base data)in a database. The data entered by the user may be converted to a singlerecord dataset that matches the format of the records of the base data.A deletion neighborhood may then be created from the (converted) usersearch data using the same rules used to create deletion neighborhoodsfor the base data. Misspellings or other mistakes in both the searchform and the base data may be compensated by enabling/specifyingstring-level edit distance. In this example use case, it is not arequirement that the user fill out all fields because the field-leveldeletion neighborhood edit distance can be adjusted to compensate formissing entries.

As an example, a public records header may be searched to find matching(or near matching records). The header may include five fields (e.g.,First Name, Last Name, Phone Number, Street Address, City) and a usermay be required to enter two or more of five presented search fieldscorresponding to the fields of the header. In this case, the field-levelmaximum edit distance would be 5−2=3. As in the previous use case,deletion neighborhood indexes may be used to rollup records based onhash values, join each ID with the hash index, and join the hash withthe ID index to resolve cases such as: A is related to B, B is relatedto C, therefore A is related to C. Once the deletion neighborhood iscreated, search code may be the same no matter how many fields areinvolved.

We claim:
 1. A method of identifying and de-duplicating records in adatabase containing a plurality of records, each record including aunique record identifier and a plurality of fields, each field capableof containing a field value, the method comprising: for each record ofthe plurality of records, determining a field-level deletionneighborhood for two or more field combinations of the record and foreach field value, the determining of the field-level deletionneighborhood comprising: determining field hash values; creating fieldpermutations; omitting at least one field for each permutation of thefield permutations; determining combined record hash values for eachpermutation; associating each record hash value to a unique recordidentifier; and determining one or more field string fragments; for eachrecord; searching other records and determining matching combined recordhash values; and assigning one or more of a unique identifier or aduplicate identifier to the other records having the matching combinedrecord hash values.
 2. The method of claim 1, further comprisingremoving, from the database, at least one of the other records havingthe duplicate identifier.
 3. The method of claim 2, further comprisingsaving at least one unique field value of the removed records in a datapreserving field identified by at least the unique identifier
 4. Themethod of claim 1, further comprising removing, from the database, allbut one candidate record from records having a same assigned uniqueidentifier.
 5. The method of claim 4, wherein the candidate recordcomprises a record having a highest accuracy confidence level.
 6. Themethod of claim 1, wherein the two or more field combinations of therecord comprise one of more of: predetermined fields, required fields,combined fields, and ordered fields.
 7. The method of claim 6, whereinthe required fields include city and state fields.
 8. The method ofclaim 1, further comprising determining a field string-level deletionneighborhood for each field value.
 9. A deduplication system,comprising: a database containing a plurality of records, each recordincluding a unique identifier and a plurality of fields, each fieldcapable of containing a field value; one or more processors; and memoryin communication with the one or more processors and storinginstructions that, when executed by the one or more processors, causethe system to: for each record of the plurality of records, determine afield-level deletion neighborhood for two or more field combinations ofthe record and for each field value, the field-level deletionneighborhood is determined by: computing field hash values; creatingfield permutations; omitting at least one field for each permutation ofthe field permutations; computing combined record hash values for eachpermutation; associating each record hash value to a unique identifier;and determining one or more field string fragments; for each record;searching other records for matching combined record hash values; andassigning a duplicate identifier to the other records having matchingcombined record hash values.
 10. The deduplication system of claim 9,wherein the instructions further cause the system to remove, from thedatabase, at least one of the other records having a duplicateidentifier.
 11. The deduplication system of claim 10, wherein theinstructions further cause the system to save at least one unique fieldvalue of removed records in a data preserving field identified by atleast the unique identifier.
 12. The deduplication system of claim 9,wherein the instructions further cause the system to assign the uniqueidentifier to the other records having the matching combined record hashvalues.
 13. The deduplication system of claim 9, wherein the two or morefield combinations of the record comprise one or more of: predeterminedfields, required fields, combined fields, and ordered fields.
 14. Thededuplication system of claim 9, wherein the two or more fieldcombinations of the record comprise required fields that include cityand state fields.
 15. A method of finding records in a database usingincomplete search criteria, the database containing a plurality ofrecords, each record including a unique identifier and a plurality offields, each field capable of containing a field value, the methodcomprising: for each record of the plurality of records, determining afield-level deletion neighborhood for two or more field combinations ofthe record and for each field value, the determining the field-leveldeletion neighborhood comprises: determining field hash values; creatingfield permutations; omitting at least one field for each permutation ofthe field permutations; determining combined record hash values for eachpermutation; associating each record hash value to a unique identifier;and determining one or more field string fragments; for each record;searching other records for matching combined record hash values; andassigning the unique identifier to the other records having the matchingcombined record hash values.
 16. The method of claim 15, wherein the twoor more field combinations of the record include one or more of:predetermined fields, required fields, combined fields, and orderedfields.
 17. The method of claim 15, wherein creating the fieldpermutations comprise omitting at least one field from each permutation.18. The method of claim 15, further comprising determining a fieldvalue-level deletion neighborhood for each field value.
 19. The methodof claim 15, wherein the two or more field combinations of the recordcomprise required fields that include city and state fields.
 20. Themethod of claim 15, wherein the two or more field combinations of therecord comprise required fields that include first name and last namefields.